\section{Conclusion}
\label{sec:conclusion}

We have introduced ASlib, a benchmark library for algorithm selection, a rapidly
growing field of research with substantial impact on various sub-communities in
artificial intelligence. Release version 1.0.1 of the library comprises 12 algorithm selection
scenarios from six different areas with a focus on (but not a limitation to) constraint satisfaction problems.
We discussed the format of new algorithm selection scenarios and showed examples
of the automated exploratory data analysis that will run for each new scenario
submitted to our online platform 
\url{http://aslib.net/}.
Finally, exploratory experiments with various simple types of algorithm
selection systems on our 12 algorithm selection scenarios demonstrated that even
simple algorithm selection systems can dramatically outperform the single
best solver and confirmed that random forest models performed best overall. 
We achieved performance improvements over the best single solver on all data sets, 
often reducing penalized average runtime by a factor of 2 and in the best case by a factor of 3.

\hh{
ASlib facilitates research on algorithm selection methods by 
providing a common set of benchmarks and tools for working with these. Similar to solver competitions, 
it enables principled comparative empirical performance assessment.
It also considerably lowers the otherwise rather high barrier for researchers to work on algorithm selection,
since anyone using the benchmark scenarios we provide does not have to perform
actual runs of the solvers contained in them.
Since our library provides performance data for the solvers and problem instances
included in each selection scenario 
(which otherwise would have to be produced, at considerable computational cost, by anyone
working with that scenario), using ASlib also substantially reduces the computational burden of performance assessments.
The carefully selected set of scenarios included in release version 1.0.1 of ASlib challenge algorithm selection 
methods in various ways and thus provide a solid basis for developing and assessing such methods.
Future updates will ensure that ASlib remains useful as research on algorithm selection progresses.
}
	
	
\subsection*{Acknowledgements}
\label{sec:ack}

%\note{FH}{I believe it is important to put in a thank-you like this to algorithm developers. We continue to steal their thunder and should most certainly share the glory with them.}
%\note{HH}{Agreed.}

We thank the creators of the algorithms and instance distributions used in our various algorithm selection scenarios. 
The performance of algorithm selection systems depends critically upon the ingenuity and tireless efforts of domain experts who continue to invent novel solver strategies.

